Fix rendering of template fields with start from trigger by dabla · Pull Request #53071 · apache/airflow

dabla · 2025-07-09T07:05:22Z

This PR fixes the issue described in #53063 in which template fields aren't rendered before being assigned to the trigger. This is because before the start_from_trigger mechanism existed, a deferred operator would always run in the worker first, which by default, would have rendered the templated fields before deferring the task.

closes: #53063

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

…can be part of the AbstractOperator and moved rendering logic from expand_start_trigger_args to _do_render_template_fields so that all the rendering logic is common in on method

…flow-core

…ask method of TaskInstance

…xpand_start_trigger_args method of BaseOperator

airflow-core/src/airflow/triggers/base.py

airflow-core/tests/unit/models/test_baseoperator.py

…erSupervisor

# Conflicts: # airflow-core/src/airflow/models/baseoperator.py # airflow-core/src/airflow/models/mappedoperator.py # airflow-core/src/airflow/serialization/serialized_objects.py # airflow-core/tests/unit/models/test_baseoperator.py # airflow-core/tests/unit/models/test_dagrun.py # airflow-core/tests/unit/serialization/test_dag_serialization.py

akshayvijayjain · 2025-07-31T06:07:26Z

@dabla can someone else take forward your work by resolving conflicts and conversations as they come

dabla · 2025-08-19T12:28:02Z

Tests fail

FAILED task-sdk/tests/task_sdk/docs/test_public_api.py::test_airflow_sdk_exports_exist - AssertionError: Missing exports in airflow.sdk: ['StartTriggerArgs']
assert not ['StartTriggerArgs']
FAILED task-sdk/tests/task_sdk/docs/test_public_api.py::test_lazy_imports_match_public_api - AssertionError: __lazy_imports missing entries for: ['StartTriggerArgs']
assert not {'StartTriggerArgs'}

@eladkal Test are fixed

ashb · 2025-08-20T12:32:17Z

Moving this to 3.1 as it's never worked, so this is a new feature, not a bug fix

uranusjr

Not sure if main would work if I merge this after #54646. Merging main in to make sure first.

airflow-core/src/airflow/triggers/base.py

task-sdk/docs/api.rst

task-sdk/src/airflow/sdk/__init__.py

This reverts commit 4362c25 from apache#53071. The original change violated Airflow's architectural separation by making the Triggerer supervisor process load DAGs and attempt template rendering. This breaks with mapped operators that use lazy sequences requiring SUPERVISOR_COMMS, which only exists in task execution context. The Triggerer should operate purely on trigger database records and pre-computed trigger kwargs, not attempt runtime DAG operations. Fixes triggerer failures with mapped operators using XCom dependencies.

…5037) This reverts commit 4362c25 from #53071. The original change violated Airflow's architectural separation by making the Triggerer supervisor process load DAGs and attempt template rendering. This breaks with mapped operators that use lazy sequences requiring SUPERVISOR_COMMS, which only exists in task execution context. The Triggerer should operate purely on trigger database records and pre-computed trigger kwargs, not attempt runtime DAG operations. Fixes triggerer failures with mapped operators using XCom dependencies.

) * refactor: Render templated fields before starting from trigger * refactor: Moved StartTriggerArgs from airflow-core to task-sdk so it can be part of the AbstractOperator and moved rendering logic from expand_start_trigger_args to _do_render_template_fields so that all the rendering logic is common in on method * refactor: Import missing datetime * refactor: Fixed import of datetime * refactor: Reformatted files * refactor: Changed import of StartTriggerArgs from new location in airflow-core * refactor: Moved call of render_template_fields from DagRun to defer_task method of TaskInstance * refactor: Moved call of render_template_fields from TaskInstance to expand_start_trigger_args method of BaseOperator * refactor: Do the template rendering in update_triggers of TriggerRunnerSupervisor * refactor: Fixed imports tests * refactor: Fixed imports * refactor: Reformatted update_triggers * refactor: Reformatted expand_start_trigger_args * refactor: Revert move of StartTriggerArgs to airflow-sdk * Revert "refactor: Revert move of StartTriggerArgs to airflow-sdk" This reverts commit 5918847. * refactor: Defined StartTriggerArgs as deprecated classes in old import location * refactor: Moved definition of StartTriggerArgs as deprecated classes in old import location to base instead of init * refactor: Moved import of StartTriggerArgs in type checking block * refactor: Added StartTriggerArgs in __all__ * Revert "refactor: Added StartTriggerArgs in __all__" This reverts commit 2e2721b. * Revert "refactor: Moved definition of StartTriggerArgs as deprecated classes in old import location to base instead of init" This reverts commit 50a8a91. * refactor: Import StartTriggerArgs in type checking block * refactor: Moved definition of StartTriggerArgs as deprecated classes in old import location to base instead of init * refactor: Fixed inner import of StartTriggerArgs in expand_start_trigger_args * refactor: Use explicit re-export for StartTriggerArgs * Revert "refactor: Moved definition of StartTriggerArgs as deprecated classes in old import location to base instead of init" This reverts commit 8a4e0c6. * Revert "refactor: Use explicit re-export for StartTriggerArgs" This reverts commit b14cb93. * Revert "refactor: Import StartTriggerArgs in type checking block" This reverts commit c1533ac. * refactor: Removed StartTriggerArgs from deprecated_classes and override getattr in base module instead * refactor: Updated warning message * refactor: Reformatted warning message * refactor: Reformatted base * refactor: Reformatted getattr to not do inner import to avoid static check error * refactor: Try mocking DagBag for test_trigger_lifecycle * refactor: Reorganized imports * refactor: Try to fix test_trigger_create_race_condition_38599 * refactor: Try to fix has not been assigned to a DAG yet error in test_trigger_create_race_condition_38599 * refactor: Try fixing DagRun validation errors for start_date and run_type in test_trigger_create_race_condition_38599 * refactor: expand_start_trigger_args method doesn't have session parameter anymore * refactor: Try fixing remaining tests * refactor: Reformatted test trigger job * refactor: Check if start_trigger_args is not None * refactor: Removed old imports * refactor: Fixed import issues * refactor: Reformatted test_trigger_create_race_condition_38599 * refactor: Try to fix test_update_triggers_prevents_duplicate_creation_queue_entries * refactor: Reformatted test_update_triggers_prevents_duplicate_creation_queue_entries * refactor: Added StartTriggerArgs to api.rst * refactor: Added StartTriggerArgs in __init__.pyi * refactor: Added lazy import for StartTriggerArgs * refactor: Changed order in __all__ for StartTriggerArgs * refactor: Reorganized imports for MappedOperator --------- Co-authored-by: David Blain <david.blain@infrabel.be> Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

…ache#55037) This reverts commit 4362c25 from apache#53071. The original change violated Airflow's architectural separation by making the Triggerer supervisor process load DAGs and attempt template rendering. This breaks with mapped operators that use lazy sequences requiring SUPERVISOR_COMMS, which only exists in task execution context. The Triggerer should operate purely on trigger database records and pre-computed trigger kwargs, not attempt runtime DAG operations. Fixes triggerer failures with mapped operators using XCom dependencies.

…5037) This reverts commit 4362c25e5cb96e3b6c99497b682d4dfe23a86aba from apache/airflow#53071. The original change violated Airflow's architectural separation by making the Triggerer supervisor process load DAGs and attempt template rendering. This breaks with mapped operators that use lazy sequences requiring SUPERVISOR_COMMS, which only exists in task execution context. The Triggerer should operate purely on trigger database records and pre-computed trigger kwargs, not attempt runtime DAG operations. Fixes triggerer failures with mapped operators using XCom dependencies. GitOrigin-RevId: 5d1d9567a24d3ff2b20eb153627d11e7d73fa7b3

This reverts commit 4362c25 from apache#53071. The original change violated Airflow's architectural separation by making the Triggerer supervisor process load DAGs and attempt template rendering. This breaks with mapped operators that use lazy sequences requiring SUPERVISOR_COMMS, which only exists in task execution context. The Triggerer should operate purely on trigger database records and pre-computed trigger kwargs, not attempt runtime DAG operations. Fixes triggerer failures with mapped operators using XCom dependencies.

) * refactor: Render templated fields before starting from trigger * refactor: Moved StartTriggerArgs from airflow-core to task-sdk so it can be part of the AbstractOperator and moved rendering logic from expand_start_trigger_args to _do_render_template_fields so that all the rendering logic is common in on method * refactor: Import missing datetime * refactor: Fixed import of datetime * refactor: Reformatted files * refactor: Changed import of StartTriggerArgs from new location in airflow-core * refactor: Moved call of render_template_fields from DagRun to defer_task method of TaskInstance * refactor: Moved call of render_template_fields from TaskInstance to expand_start_trigger_args method of BaseOperator * refactor: Do the template rendering in update_triggers of TriggerRunnerSupervisor * refactor: Fixed imports tests * refactor: Fixed imports * refactor: Reformatted update_triggers * refactor: Reformatted expand_start_trigger_args * refactor: Revert move of StartTriggerArgs to airflow-sdk * Revert "refactor: Revert move of StartTriggerArgs to airflow-sdk" This reverts commit 5918847. * refactor: Defined StartTriggerArgs as deprecated classes in old import location * refactor: Moved definition of StartTriggerArgs as deprecated classes in old import location to base instead of init * refactor: Moved import of StartTriggerArgs in type checking block * refactor: Added StartTriggerArgs in __all__ * Revert "refactor: Added StartTriggerArgs in __all__" This reverts commit 2e2721b. * Revert "refactor: Moved definition of StartTriggerArgs as deprecated classes in old import location to base instead of init" This reverts commit 50a8a91. * refactor: Import StartTriggerArgs in type checking block * refactor: Moved definition of StartTriggerArgs as deprecated classes in old import location to base instead of init * refactor: Fixed inner import of StartTriggerArgs in expand_start_trigger_args * refactor: Use explicit re-export for StartTriggerArgs * Revert "refactor: Moved definition of StartTriggerArgs as deprecated classes in old import location to base instead of init" This reverts commit 8a4e0c6. * Revert "refactor: Use explicit re-export for StartTriggerArgs" This reverts commit b14cb93. * Revert "refactor: Import StartTriggerArgs in type checking block" This reverts commit c1533ac. * refactor: Removed StartTriggerArgs from deprecated_classes and override getattr in base module instead * refactor: Updated warning message * refactor: Reformatted warning message * refactor: Reformatted base * refactor: Reformatted getattr to not do inner import to avoid static check error * refactor: Try mocking DagBag for test_trigger_lifecycle * refactor: Reorganized imports * refactor: Try to fix test_trigger_create_race_condition_38599 * refactor: Try to fix has not been assigned to a DAG yet error in test_trigger_create_race_condition_38599 * refactor: Try fixing DagRun validation errors for start_date and run_type in test_trigger_create_race_condition_38599 * refactor: expand_start_trigger_args method doesn't have session parameter anymore * refactor: Try fixing remaining tests * refactor: Reformatted test trigger job * refactor: Check if start_trigger_args is not None * refactor: Removed old imports * refactor: Fixed import issues * refactor: Reformatted test_trigger_create_race_condition_38599 * refactor: Try to fix test_update_triggers_prevents_duplicate_creation_queue_entries * refactor: Reformatted test_update_triggers_prevents_duplicate_creation_queue_entries * refactor: Added StartTriggerArgs to api.rst * refactor: Added StartTriggerArgs in __init__.pyi * refactor: Added lazy import for StartTriggerArgs * refactor: Changed order in __all__ for StartTriggerArgs * refactor: Reorganized imports for MappedOperator --------- Co-authored-by: David Blain <david.blain@infrabel.be> Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

refactor: Render templated fields before starting from trigger

498d19d

dabla requested review from XD-DENG, ashb and uranusjr as code owners July 9, 2025 07:05

dabla marked this pull request as draft July 9, 2025 07:05

Merge branch 'main' into fix/templated-fields-with-start-from-trigger

9fb1303

dabla mentioned this pull request Jul 9, 2025

Allow MSGraphAsyncOperator and MSGraphSensor to directly start from trigger instead of worker #53009

Draft

davidblain-infrabel and others added 4 commits July 9, 2025 12:26

refactor: Moved StartTriggerArgs from airflow-core to task-sdk so it …

34aa1fa

…can be part of the AbstractOperator and moved rendering logic from expand_start_trigger_args to _do_render_template_fields so that all the rendering logic is common in on method

Merge branch 'main' into fix/templated-fields-with-start-from-trigger

bac3be7

refactor: Import missing datetime

4fd077d

refactor: Fixed import of datetime

0b2ac51

dabla marked this pull request as ready for review July 9, 2025 10:58

dabla requested review from amoghrajesh and kaxil as code owners July 9, 2025 10:58

davidblain-infrabel added 2 commits July 9, 2025 13:27

refactor: Reformatted files

a68bcb0

refactor: Changed import of StartTriggerArgs from new location in air…

585d2b8

…flow-core

dabla requested a review from bolkedebruin as a code owner July 9, 2025 11:56

dabla and others added 3 commits July 9, 2025 13:56

Merge branch 'main' into fix/templated-fields-with-start-from-trigger

c43f152

refactor: Moved call of render_template_fields from DagRun to defer_t…

87325dc

…ask method of TaskInstance

refactor: Moved call of render_template_fields from TaskInstance to e…

42bc711

…xpand_start_trigger_args method of BaseOperator

ashb reviewed Jul 9, 2025

View reviewed changes

airflow-core/src/airflow/triggers/base.py Outdated Show resolved Hide resolved

airflow-core/tests/unit/models/test_baseoperator.py Outdated Show resolved Hide resolved

dabla marked this pull request as draft July 10, 2025 06:31

davidblain-infrabel added 2 commits July 14, 2025 11:31

refactor: Do the template rendering in update_triggers of TriggerRunn…

897d7c0

…erSupervisor

dabla mentioned this pull request Jul 23, 2025

StartFromTrigger doesn't work with templated fields #53063

Open

2 tasks

eladkal added this to the Airflow 3.0.4 milestone Jul 30, 2025

eladkal added type:bug-fix Changelog: Bug Fixes backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch labels Jul 30, 2025

Merge branch 'main' into fix/templated-fields-with-start-from-trigger

80ed0f1

dabla mentioned this pull request Aug 20, 2025

Disable start_from_trigger altogether for now #54646

Merged

ashb modified the milestones: Airflow 3.0.6, Airflow 3.1.0 Aug 20, 2025

ashb removed the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Aug 20, 2025

Merge branch 'main' into fix/templated-fields-with-start-from-trigger

e9fa7f3

uranusjr approved these changes Aug 20, 2025

View reviewed changes

uranusjr reviewed Aug 20, 2025

View reviewed changes

airflow-core/src/airflow/triggers/base.py Show resolved Hide resolved

uranusjr reviewed Aug 20, 2025

View reviewed changes

task-sdk/docs/api.rst Show resolved Hide resolved

uranusjr reviewed Aug 20, 2025

View reviewed changes

task-sdk/src/airflow/sdk/__init__.py Show resolved Hide resolved

dabla added 3 commits August 21, 2025 13:58

Merge branch 'main' into fix/templated-fields-with-start-from-trigger

12ad45d

refactor: Reorganized imports for MappedOperator

3408529

Merge branch 'main' into fix/templated-fields-with-start-from-trigger

da832ba

eladkal added type:new-feature Changelog: New Features and removed type:bug-fix Changelog: Bug Fixes labels Aug 22, 2025

eladkal approved these changes Aug 22, 2025

View reviewed changes

eladkal merged commit 4362c25 into apache:main Aug 22, 2025
79 checks passed

kaxil mentioned this pull request Aug 28, 2025

Revert "Fix rendering of template fields with start from trigger" #55037

Merged

dabla mentioned this pull request Aug 29, 2025

Re-enable start_from_trigger feature with rendering of template fields #55068

Open

eladkal mentioned this pull request Sep 5, 2025

Status of testing Providers that were prepared on September 05, 2025 #55285

Closed

meldiner mentioned this pull request Feb 6, 2026

Revert "Fix rendering of template fields with start from trigger" faros-ai/airflow#10

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix rendering of template fields with start from trigger#53071

Fix rendering of template fields with start from trigger#53071
eladkal merged 80 commits intoapache:mainfrom
dabla:fix/templated-fields-with-start-from-trigger

dabla commented Jul 9, 2025 •

edited by eladkal

Loading

Uh oh!

Uh oh!

Uh oh!

akshayvijayjain commented Jul 31, 2025

Uh oh!

dabla commented Aug 19, 2025 •

edited

Loading

Uh oh!

ashb commented Aug 20, 2025

Uh oh!

uranusjr left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

dabla commented Jul 9, 2025 • edited by eladkal Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

akshayvijayjain commented Jul 31, 2025

Uh oh!

dabla commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb commented Aug 20, 2025

Uh oh!

uranusjr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dabla commented Jul 9, 2025 •

edited by eladkal

Loading

dabla commented Aug 19, 2025 •

edited

Loading